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BACKGROUND OF THE INVENTION 



Field of Invention 
5 The present invention pertains to the field of 

distributed systems. More particularly, this 
invention relates to time synchronization among 
clocks in a distributed system. 

10 Art Background 

Distributed systems are commonly arranged as a 
(,* collection of nodes which are interconnected via one 

or more network communication links. These network 
communication links may be packetized links such as 
15 Ethernet or one or more of a variety of other 

packetized links that are adapted to distributed 
u control system applications. 
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Distributed systems commonly benefit from 
precise control of the timing at the distributed 
H nodes. U.S. Patent No. 5,566,180 of Eidson et . al . 

teaches a method and apparatus for providing precise 
control of timing in distributed nodes by 
synchronizing the local clocks in the distributed 
25 nodes. The synchronization protocol of Eidson et . 

al. involves the exchange of timing data packets and 
follow up packets among the nodes so that the delay 
in the transfer of a timing data packet from a first 
node to a second node in combination with timing 
information in a follow up packet can be used to 
accurately adjust a local clock in the second node. 

A variety of conditions that are commonly found 
in distributed systems may introduce variation or 
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jitter in the delay in the transfer of a timing data 
packet. For example, communication circuitry at 
various points in the distributed system may 
introduce jitter. In addition, communication 
circuits such as gateways can introduce jitter that 
depends on the volume of traffic in the system. 
Unfortunately , such jitter may reduce the accuracy of 
time synchronization in a distributed system. 
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SUMMARY OF THE INVENTION 



A distributed system is disclosed with a timing 
signal path for increased precision in time 
synchronization among distributed system clocks. A 
distributed system according to the present teachings 
includes a master clock coupled to a timing signal 
path and a set of slave clocks coupled to the timing 
signal path. The master clock includes means for 
generating a timing signal on the timing signal path 
in response to time events associated with the master 
clock. The slave clocks include means for adjusting 
their local time in response to the timing signal 
received via the timing signal path. Additional 
timing information may be exchanged among the master 
and slave clocks using a network. 

Other features and advantages of the present 
invention will be apparent from the detailed 
description that follows. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



The present invention is described with respect 
to particular exemplary embodiments thereof and 
reference is accordingly made to the drawings in 
which: 

Figure 1 shows a distributed system according to 
the present teachings; 

Figure 2 shows another distributed system 
according to the present teachings; 

Figure 3 shows an example implementation of a 
clock node which may be configured as a master clock 
or a slave clock; 

Figure 4 shows another example implementation of 
a clock node which may be configured as a master 
clock; 

Figure 5 shows yet another distributed system 
according to the present teachings. 
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DETAILED DESCRIPTION 



Figure 1 shows a distributed system 3 0 according 
to the present teachings. The distributed system 30 
5 includes a master clock 10 and a set of slave clocks 
12-14. The master clock 10 and the slave clocks 12- 
14 maintain synchronization using information 
transferred via a network 2 0 and a timing signal path 
22 . 

10 

The master clock 10 and the slave clocks 12-14 
may be implemented as nodes that communicate via the 
O network 2 0 when performing application-specific 

y functions. The nodes that contain the master clock 

15 io and the slave clocks 12-14 may include 
U hardware /software elements that perform application- 

u specific functions associated with the distributed 

jU system 3 0 as well as hardware/elements for clock 

signal generation and time synchronization according 
£ 2 0 to the present teachings. 

The master clock 10 and the slave clocks 12-14 
implement hardware/software elements for maintaining 
synchronization of their local time values by 

25 transferring timing messages via the network 20. In 
one embodiment, the master clock 10 and the slave 
clocks 12-14 perform a synchronization protocol 
described in U.S. Patent No. 5,566,180 of Eidson et . 
al . In general, the synchronization protocol enables 

30 the selection of any of the clocks 10-14 as the 
master clock. 
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The master clock 10 via its T 0UT port 
periodically issues a timing signal 44 on the timing 
signal path 22. The timing signal 44 is aligned to a 
time event associated with the master clock 10. For 
5 example, the time event may be a seconds boundary of 
the master clock 10. The leading edge or the 
trailing edge of the pulse in the timing signal 44 or 
the middle of the pulse in the timing signal 44 may 
be aligned in time with a change in the seconds value 
10 of the master clock 10. In another example, the time 
event may be an every other second boundary of the 
y, master clock 10. The master clock 10 generates a 

time-stamp when the time event occurs - for example 
at the seconds boundary of the master clock 10. 

15 

The slave clocks 12-14 each time stamp their 
receipt of the timing signal 44. For example, the 
slave clock 12 generates a time-stamp when the timing 
signal 44 is received via its corresponding T IN port. 
2 0 The time- stamp may be generated upon detection of the 
leading or trailing edge of the pulse in the timing 
signal 44 or at some other point in the timing signal 
as appropriate. 

25 The master clock 10 then issues a follow-up 

message 4 0 via the network 2 0 that includes a time- 
stamp 42. The time-stamp 42 is the local time in the 
master clock 10 that the time event occurred in the 
master clock 10. Each slave clock 12-14 receives the 
30 message 40, extracts the time-stamp 42, and uses its 
corresponding time-stamp obtained upon receipt of the 
timing signal 44 along with the time- stamp 42 to 
adjust its local time value. 
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For example, the relative difference between the 
time-stamp 42 and the time-stamp obtained by the 
slave clock 12 on receipt of the timing signal 44 
indicates the relative error between the local time 
of the master clock 10 and the local time of the 
slave clock 12 . The slave clock 12 includes 
circuitry for speeding up or slowing down or 
reloading its local time value in response to this 
relative error. 

In an alternative embodiment, the master clock 
10 encodes the time-stamp 42 and transmits it to the 
slave clocks 12-14 via the timing signal path 22. 
Each slave clock 12-14 decodes the time-stamp 42 
obtained via the timing signal path 22 and uses it in 
its clock adjustment calculations. 

A propagation delay exists during transfer of 
the timing signal 44 from the master clock 10 to the 
slave clocks 12-14. The propagation delay to the 
slave clocks 12-14 may differ depending on the 
topology of the distributed system 30. In one 
embodiment, the propagation delay of the timing 
signal 44 is uniquely determined for each slave clock 
12-14 - for example through measurement or 
calculation. Each slave clock 12-14 performs the 
appropriate correction to its time synchronization 
calculations in response to its propagation delay. 

Alternatively, each slave clock 12-14 may 
determine its propagation delay along the timing 
signal path 22 by transferring a timing signal back 
to the master clock 10 while the transmit and receive 
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times of the timing signal are measured. For 
example, the slave clock 12 generates a timing signal 
and a time-stamp for the timing signal, and transfers 
the timing signal via the timing signal path 22. The 
5 master clock 10 receives the timing signal via the 
timing signal path 22, generates a time-stamp in 
response to the timing signal, and transmits the 
time-stamp via the network 20 in a message. The 
slave clock 12 obtains the time- stamp from the master 
10 clock 10 via the network 20 and uses it together with 
the time -stamp it generated on transmission of the 
timing signal to calculate the propagation delay from 
* m the slave clock 12 to the master clock 10. 

■5= -2 

t~ ■ 4 

jyS 15 The timing signal path 2 2 may be implemented 

'f k with a multi-drop bus or point-to-point communication 

Q links. One example of a multi-drop bus is an RS-485 

f bus . 

PJ 

l \% 2 0 The network 2 0 may be a packet -based 

fj communication network such as Ethernet. Another 

example of a packet -based network is LonTalk which is 
adapted to distributed control environments. 



25 Figure 2 shows a distributed system 60 according 

to the present teachings. The distributed system 60 
includes a master clock 62 and a set of slave clocks 
70-74 which communicate via a network 64. 



30 The master clock 62 issues a set of timing 

signals 90-94 to the slave clocks 70-74 via its T 0UT 
ports. The timing signals 90-94 are similar to the 
timing signal 44, i.e. are aligned to a timing event 
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of the master clock 62. The master clock 62 
generates a time-stamp when the time event occurs. 
Each slave clock 70-74 generates a corresponding 
time-stamp when the corresponding timing signal 90-94 
5 is received via its T IN port. 

The master clock 62 then issues a follow-up 
message 80 via the network 64 that includes a time- 
stamp 82. The time-stamp 82 indicates the local time 
in the master clock 62 that the time event occurred. 
Each slave clock 70-74 uses its corresponding time- 
stamp obtained upon receipt of the corresponding 
timing signal 90-94 along with the time-stamp 82 to 
determine an adjustment to its local time value. 

The master clock 62 may be synchronized to a GPS 
receiver (not shown) or to some other precise time 
source . 

The ports on the slave clocks 70-74 may be 
similar to the ports on the slave clocks 12-14, i.e. 
an RS-485 port which is set in the receive mode. 

The connecting cables between the T 0UT ports of 
2 5 the master clock 62 and the T IN ports of the slave 
clocks 70-74 may have calibrated lengths to reduce 
errors caused by differing propagation delays in the 
timing signals 90-94. The slave clocks 70-74 may 
correct for propagation delays as described above. 

30 

Figure 3 shows an example implementation of a 
clock node 100 which may be configured as the master 
clock 10 or any of the slave clocks 12-14. The clock 
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node 100 includes a timing physical interface circuit 
(PHY) 112 for communication via the timing signal 
path 22 and a network PHY 114 for communication via 
the network 20. If the clock node 100 is a master 
clock then a node microprocessor 110 uses an XMIT 
signal to cause the timing PHY 112 to enter a 
transmit mode. If the clock node 10 0 is a slave 
clock then the node microprocessor 110 uses the XMIT 
signal to cause the timing PHY 112 to enter a receive 
mode . 

In transmit (master) mode, the transmit input 
(Tx) to the timing PHY 112 is generated by a 
comparator 118. The comparator 118 compares a 
current local time obtained from a clock counter 13 0 
to a time value held in a time-bomb latch 116. The 
time-bomb latch 116 is set by the node microprocessor 
110 to generate the timing signal 44 at a desired 
time event - for example every other second of the 
clock counter 130. If the delay from the comparator 
118 to the timing signal path 22 is known, for 
example through design or measurement, then the value 
in the time-bomb latch 116 may be adjusted 
accordingly. 

Alternatively, the timing signal 44 may be 
derived using a gate, enabled by the node 
microprocessor 110, from the fractional second 
rollover bit or the units seconds bit, etc. of the 
clock counter 130. 

In receive (slave) mode, the timing PHY 112 
transfers a received signal (Rx) to an edge detector 

Attorney Docket No. 10003336 



120 which detects a leading or trailing edge of the 
timing signal 44. An edge detect signal from the 
edge detector 120 causes a time-stamp latch 122 to 
latch a local time value from the clock counter 13 0. 
A timing packet recognizer (TPR) 124 and a time-stamp 
latch 12 6 are used to detect and time stamp timing 
packets received via the network 20. The latched 
time values from the time-stamp latches 122 and 126 
are provided to the node microprocessor 110 to be 
used in calculating corrections to be applied to the 
time value in the clock counter 130. 

The node microprocessor 110 may detect the 
presence or absence of timing signals by determining 
whether the time stamp latch 122 is ever updated. 
Alternatively, the Rx signal may set a flip-flop 
which is read and reset by the node microprocessor 
110 to detect the presence or absence of timing 
signals . 

Figure 4 shows an example implementation of a 
clock node 14 0 which may be configured as the master 
clock 62. The clock node 140 includes a timing PHY 
146 for providing the timing signal 90 to the slave 
clock 70 and a network PHY 144 for communication via 
the network 64. The clock node 14 0 includes a set of 
additional output ports 148 for providing the timing 
signals 92-94 to the slave clocks 72-74. 

The transmit input (Tx) to the timing PHY 14 6 is 
generated by a comparator 150 which compares a 
current local time value obtained from a clock 
counter 160 to a time value held in a time-bomb latch 
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152. The time-bomb latch 152 is set by a node 
microprocessor 142 to generate the timing signals 90- 
94 at desired time events. The signal generated by 
comparator 15 0 that indicates a match is provided to 
5 a clock distribution circuit 154. The clock 

distribution circuit 154 provides multiple Tx output 
signals with relatively low timing skew to be 
distributed as the timing signals 90-94. 

Figure 5 shows a distributed system 2 50 
according to the present teachings. The distributed 
system 250 includes a master clock 200 and a slave 
clock 202. The master clock 200 includes a signal 
generator 210 which generates and transmits a 
continuous frequency timing signal to the slave clock 
202 via a timing signal path 220. The timing signal 
path 220 may be a multi-drop bus such as RS-485 or 
may be a point-to-point bus. 

The continuous frequency timing signal generated 
by the signal generator 210 may be equal to or 
proportional to the frequency of the master clock 
200. For example, the signal generator 210 may 
derive a 10 MHz timing signal from a local 
oscillator . 

The slave clock 2 02 includes a phase lock loop 
circuit 212 which phase locks to the timing signal 
received on the timing signal path 220 to yield a 
3 0 local clock frequency. The phase locking of the 
local clock frequency in the slave clock 2 02 to a 
continuous timing signal derived from the local clock 
signal in the master clock 2 00 provides 
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syntonization, i.e. matching frequencies, among the 
master clock 200 and the slave clock 202. The timing 
signal path 22 0 may be extended to other slave clocks 
in the distributed system 250 having phase lock loop 
circuits . 

The continuous frequency timing signal on the 
timing signal path 22 0 may include a normal pattern 
for phase locking and a periodic distinguished 
pattern that indicates a time event in the master 
clock 200 - for example one second boundaries of the 
master clock 200. The slave clock 202 may include 
bit pattern recognition circuitry for recognizing the 
receipt of the distinguished pattern and time-stamp 
circuitry for time stamping the receipt of the 
distinguished pattern. The master clock 2 00 may send 
a follow-up message via a network 222 that includes a 
time-stamp indicating the local time in the master 
clock 200 of transmission of the distinguished 
pattern. The slave node 202 uses the time-stamps to 
determine corrections to its local time in a manner 
previously described. 

In one embodiment, the continuous frequency 
timing signal is Manchester encoded in which the 
normal pattern is alternating ones and zeros and the 
distinguished pattern is two consecutive ones. 

In some embodiments, the time-stamp that would 
otherwise be sent to the slave clock 2 02 via the 
network 222 may instead be encoded and transmitted to 
the slave clock 202 via the timing signal path 220. 
The time -stamp may be, for example, the UTC time in 
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terms of the master clock 200 of the time event 
corresponding to the distinguished pattern. The 
slave clock 2 02 includes circuitry for decoding the 
received time-stamp. 

The foregoing detailed description of the 
present invention is provided for the purposes of 
illustration and is not intended to be exhaustive or 
to limit the invention to the precise embodiment 
disclosed. Accordingly, the scope of the present 
invention is defined by the appended claims. 
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